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Abstract 


This chapter describes a Copernicus Access Platform Intermediate Layers Small- 
Scale Demonstrator, which is a general platform for the handling, analysis, and 
interpretation of Earth observation satellite images, mainly exploiting big data of 
the European Copernicus Programme by artificial intelligence (AI) methods. From 
2020, the platform will be applied at a regional and national level to various use 
cases such as urban expansion, forest health, and natural disasters. Its workflows 
allow the selection of satellite images from data archives, the extraction of useful 
information from the metadata, the generation of descriptors for each individual 
image, the ingestion of image and descriptor data into a common database, the 
assignment of semantic content labels to image patches, and the possibility to 
search and to retrieve similar content-related image patches. The main two compo- 
nents, namely, data mining and data fusion, are detailed and validated. The most 
important contributions of this chapter are the integration of these two components 
with a Copernicus platform on top of the European DIAS system, for the purpose 
of large-scale Earth observation image annotation, and the measurement of the 
clustering and classification performances of various Copernicus Sentinel and 
third-party mission data. The average classification accuracy is ranging from 80 to 
95% depending on the type of images. 


Keywords: Earth observation, machine learning, data mining, Copernicus 
Programme, TerraSAR-X 


1. Introduction 


Typical shortcomings of current image analysis tools are the lack of content 
understanding. This becomes apparent with current developments in Earth obser- 
vation and data analysis [1]. In this chapter, we therefore concentrate on artificial 
intelligence (AI) applications and our solution strategies as our main objectives 
in the field of remote sensing, i.e., the acquisition and semantic interpretation of 
instrument data from remote platforms such as aircraft or satellites observing, for 
instance, atmospheric phenomena on Earth for weather prediction—or icebergs 
drifting in arctic waters endangering maritime transport. In particular, we will 
describe the exploitation of imaging data acquired by Earth-observing satellites and 
their sensors. 
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These satellites may either circle about the Earth (mostly on low polar Earth 
orbits) or be operated from stationary or slowly moving points high above our 
planet (on so-called geostationary or geosynchronous orbits). Typical examples 
are Earth-observing and meteorological satellites. All these instruments have been 
designed with dedicated goals that, as a rule, can only be fulfilled by systematic and 
interactive data processing and data interpretation on the ground. The processing 
and data analysis chains are then the main candidates where one can and shall apply 
modern data science approaches (e.g., machine learning and artificial intelligence) 
in order to fully exploit the full information content of the sensor data. 

In general, we have quite a number of different sensors installed on satellites. These 
include passive instruments observing the backscattered solar illumination or thermal 
emissions from the Earth—or active imaging instruments (transmitting and receiv- 
ing light pulses or radio signals toward and from the target area being observed). For 
the ease of understanding, we will limit ourselves to optical sensors operating in the 
visible and infrared spectral ranges and to radar sensors applying synthetic-aperture 
radar (SAR) concepts [2, 3]. These instruments provide large-scale images with a typi- 
cal spatial resolution of 1-40 m per pixel. The images can be acquired from spacecraft 
orbits that cover the Earth completely with well-defined repeat cycles. 

After being transmitted to the ground, the image data will have to undergo 
systematic processing steps. Typically, the processing schemes follow a stepwise 
approach where for all steps the image data are accompanied by the necessary 
descriptor data (metadata). The processing chains start with what we call level-0 
data consisting of reordered and annotated detector data; level-1 data provide 
calibrated sensor data, while level-2 data contain data in commonly known physical 
units preferably on regular spatial or map grids. Then level-3 data are higher-level 
products such as thematic maps or time series results (obtained by merging or 
concatenation of several individual images) or similar operations. Finally, users 
can apply additional interactive processing steps on their own or exploit available 
software/platform concepts [4]. 

This principle of ordered value-adding requires well-established techniques for 
data management, batch processing and databases, local and distributed (cloud) pro- 
cessing, understanding of the information flow, experience with learning principles, 
knowledge extraction from image and library data, and discovery of image seman- 
tics. At present, typical data sources with easy access are publicly available scientific 
image data provided by the European Copernicus mission with its Sentinel satellites 
[5, 6] as well as high-resolution remote sensing images [7, 8]. The European Sentinel 
satellites comprise among others a constellation of SAR imagers (i.e., Sentinel-1A/ 
Sentinel-1B providing typically large radar images, with a ground sampling distance 
of 20 meters and selectable horizontal and vertical polarizations), and a constellation 
of optical imagers (i.e., Sentinel-2A/Sentinel-2B delivering typically large multi- 
spectral images with 13 different bands and a ground resolution—depending on 
the bands—of 10-60 m). This space segment of the Copernicus mission is comple- 
mented by systematic level-1 and level-2 image data processing on the ground and by 
support environments that serve as comfortable platforms for further data handling 
and interpretation covering all aspects of applied data science. These approaches 
then pave the way for deeper semantic data analysis and understanding as typically 
required in Earth observation for crop yield predictions, atmospheric research, etc. 

The design of Earth observation (EO) missions as constellations of several satel- 
lites brings important advantages. However, this is not the case for some of the most 
popular EO missions. Figure 1 shows typical TerraSAR-X and Copernicus Sentinel 
overpasses from different orbits and their target areas. 

TerraSAR-X flies on a polar Sun-synchronous circular dawn-dusk orbit. This 
satellite shares its orbit plane with its twin satellite TanDEM-X (keeping a 97.44° 
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Figure 1. 
Satellite overpasses of Sentinel-1A /Sentinel-1B, Sentinel-2A/Sentinel-2B, and TerraSAR-X (on 23th of August 
2018 starting at 14:02 UT) [12]. 


orbital phasing difference) and a repeat cycle of 11 days with 167 orbits per cycle. 
Due to its flexibility, TerraSAR-X can cover any point on Earth within a maximum 
of 4.5 days and 90% of the Earth’s surface within 2 days [9]. 

The Sentinel-1 satellites fly on a near-polar, Sun-synchronous orbit, too. The 
satellite constellations (comprising Sentinel-1A and Sentinel-1B) share the same 
orbit plane with a 180° orbital phasing difference and a repeat cycle of 6 days with 
175 orbits per cycle. Sentinel-1 can cover the equator on 3 days, the Artic on less 
than 1 day, and Europe, Canada, and shipping routes in 1-3 days [10]. 

Like the Sentinel-1 constellation, the Sentinel-2 constellations (comprising 
Sentinel-2A and Sentinel-2B) share the same orbit with a separation of 180°. The 
repeat cycle is 5 days with 143 orbits per cycle. Sentinel-2 can cover the equator on 
5 days under cloud-free conditions and in 2-3 days at mid-latitudes [11]. 

When selecting data for fusion, we have to constrain ourselves to data acquired 
as close as possible in time. 

These data handling approaches are typical for recent advances in big data sce- 
narios in distributed systems on the web (e.g., with high data volumes and through- 
put rates, conventional and innovative data processing steps, additional necessary 
tools and environments, and greater user expectations). In our case, this affects the 
tasks of image processing (e.g., data fusion), image understanding, and compari- 
sons with physical models. This can also be seen when we look at the evolution of 
satellite data analysis. While early concepts started with data being transferred to 
algorithms, current systems often transfer data to archives, and future systems may 
support more and more distributed systems. 

A typical example is the full functionality offered by machine learning tools, 
while the basic ideas of future data science aspects for Earth observation as seen by 
the European Space Agency can be found in [13]. In our case, we are interested in 
applying more theoretical data science, machine learning, and artificial intelligence 
(for instance, deep learning, powerful classification maps, and prediction results) 
together with interactive visualization on various information levels. These ideas 
will be dealt with below for three remote sensing scenarios as detailed in [14]: 


e Urban monitoring (urban growth and sprawl, urban classification, and 
semantic indicators) 


e Quantitative interpretation of forested areas 


e Disaster monitoring (earthquakes, inundations, mud slides, etc.) 
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Here traceable products yielding quantitative data about physical phenomena, 
change maps, and change predictions are among our primary goals. Of course, we 
have to consider the implementation effort as well as the attainable accuracy of our 
products. For each scenario dealt with below, the reader should try to understand 
what the additional value of machine learning, artificial intelligence, and compre- 
hensive use of data science concepts brings about. 

The basic terms of machine learning, artificial intelligence, and data science 
shall be understood in the following sense: 


e We use the term “machine learning” mainly when we talk about learning target 
category parameters derived from selected images and applying these param- 
eters to other examples. Currently, we see much progress by “deep” techniques 
(e.g., deep learning [15, 16]). An important point is the selection of reliable 
reference data for traceable validation and verification of the methods. 


“Artificial intelligence” describes how machine learning results are exploited 
for further use. Typically this includes recognizing and being aware of typical 
situations, making decisions based on the recognized high-level parameters, 
and predicting future developments. To this end, one can profit from external 
databases complementing machine learning results. 


e “Data science” covers the entire field of comprehensive data management and 
tools, machine learning, and artificial intelligence. This includes topics like 
distributed processing, monitoring of workflows, visualization techniques, 
and performance monitoring. Even seemingly trivial tasks (e.g., accessing and 
handling of data) may belong to data science. However, remote sensing still is 
in urgent need of efficient tools to familiarize the user community with remote 
sensing opportunities. 


When we look at remote sensing in more detail, we currently see many efforts to 
transform sensor data to physical quantities that can be exploited for quantitative 
analysis or modeling. If we accomplish this, we can combine measured data with 
physical models and find quantitative parameters for predictions. 

In the following, we describe how we applied these concepts in a research project 
funded by the European Union [17]; the project’s main objective is to allow the 
creation of added value from Copernicus data through the provisioning of model- 
ing and analytics tools for data collection, processing, storage, and access that are 
provided by the Copernicus Data and Information Access Services (DIAS) [18] and 
creating a data science workflow where sub-images (image chips) are annotated, 
administered, and validated based on their assigned semantic labels [19]. 

The chapter is organized in seven main sections. Section 2 explains the 
CANDELA platform used for prototyping EO applications, while Section 3 
describes the characteristics of the data set. Section 4 presents typical examples 
which a user can obtain when using the platform from Section 2 and the data set 
from Section 3. Section 5 illustrates the perspectives in EO data science workflows 
and Section 6 summarizes our conclusions, while Section 7 contains the future 
work. The chapter ends with acknowledgments and a list of references. 


2. The CANDELA platform 


CANDELA’ main objective is the creation of additional value from Copernicus 
data through the provisioning of modeling and analytics tools provided that the 
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CANDELA platform [17]. 


tasks of data collection, processing, storage, and access will be carried out by the 
Copernicus Data Information and Access Service [18]. The corresponding flowchart 
is presented in Figure 2 and in [17]. In the end, after the integration of all compo- 
nents, CANDELA will be deployed on top of DIAS. 

The CANDELA platform [17] allows prototyping of EO applications by applying 
efficient data retrieval, data mining augmented with machine learning techniques, 
as well as interoperability in order to fully benefit from the available assets and to 
add more value to the satellite data. It also helps to interactively detect objects or 
structures and to classify land cover categories. 

The implementation of the platform is putting in place a set of powerful tools in 
artificial intelligence environments (e.g., with machine learning and deep learn- 
ing). These tools have as their objectives: 


e To process large volumes of EO data and to perform data analytics 
e To extract the information content from the EO data based on data mining 


e To fuse various EO sensors in order to increase and to complement the infor- 
mation extracted from different sensors 


e To apply deep learning to detect changes in EO data 
e To semantically search and index our EO image catalog 


From this list of objectives, we focus on two of them, namely, data mining and 
data fusion (see Figure 3). Our goal is to simplify data access and to analyze large 
volumes of EO data without specific knowledge about the processing of EO data 
and to fuse the outputs for content exploration. 

For the development of the data mining component, we started from [20], and 
we improved the cascaded active learning system of [21] for typical Copernicus 
Earth observation images. Its implementation, test, and validation aim at auto- 
mated knowledge extraction and image content interpretation. The results are 
presented in Section 4.1. 

Regarding data fusion, a new sub-component had to be developed within 
data mining. This new sub-component fuses multispectral and SAR images. 
There are two types of fusions; one is performed at the feature level and the 
other one at the semantic level. The results are shown in Section 4.2 for feature- 
level fusion. 
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Figure 3. 
Block diagram of the CANDELA platform modules [17]. 


3. Data set description for CANDELA 


Our main data sets extracted from different instruments are Earth’s surface 
images of the European Copernicus Programme (e.g., Sentinel-1 and Sentinel-2). 
Sentinel-1 is a twin satellite synthetic-aperture radar configuration, while Sentinel-2 
is also a twin satellite configuration, each carrying a multispectral imager [22, 23]. 

There are three reasons why we are selecting and using Sentinel-1 and Sentinel-2 
images. Firstly, we can recognize different target area details in overlapping radar 
and optical images complementing each other with rapid succession. Secondly, 
individually selectable Sentinel-1 and Sentinel-2 images can be rectified and 
co-aligned by publicly available toolbox routines offered by ESA allowing a straight- 
forward image comparison or image fusion. Thirdly, all Sentinel instruments are 
totally openly available to the EO community. Many publications (dedicated confer- 
ences [1, 24—26]) already describe newly discovered Earth’s surface characteristics 
derived from the individual instruments. 

Furthermore, the long-term operations of the Sentinel satellites allow the 
interpretation of image time series or even the combination of time series data 
with external supplementary data via additional data mining and data fusion tools 
[1, 25, 26]. 

Besides these data sets, we include other third-party EO mission data sets as 
specified by CANDELA users (e.g., TerraSAR-X and WorldView). 


3.1 Sentinel-1 data 


The Sentinel-1 mission comprises a constellation of two satellites (launched on 
April 1, 2014, and on April 25, 2016), operating in C-band for synthetic-aperture 
radar imaging. SAR has the advantage of operating at wavelengths not impeded by 
thin cloud cover, or a lack of solar illumination, and can acquire data over a selected 
area during day- or nighttime under nearly no weather condition restrictions. The 
repeat period of each satellite is 12 days; that means every 6 days there is an acquisi- 
tion by one of the two satellites. 

The Sentinel-1 characteristics are presented in detail in [22]. From the multitude 
of parameters/configurations that exist for Sentinel-1, we have selected as examples 
the following configurations based on data availability, the CANDELA use cases, 
and our previous experiments: level-1 Ground Range Detected (GRD) products with 
high resolution (HR) taken routinely in Interferometric Wide (I'W) swath mode. 
These products/data are produced (prior to geo-coding) with a pixel spacing of 
10 x 10 m and correspond to about five looks and a resolution (range x azimuth) of 
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20 x 22 m. They have a nearly uniform signal-to-noise ratio (SNR) and also a stable 
distributed target ambiguity ratio (DTAR). For these products, the data are provided 
in dual polarization, VV and VH for land and HH and HV for polar target areas. 


3.2 Sentinel-2 data 


The Sentinel-2 mission (like Sentinel-1) comprises a constellation of two satel- 
lites (launched on June 23, 2015, and on March 7, 2017) able to collect multispectral 
data and is affected by the weather conditions (e.g., cloud cover). The repeat period 
of each satellite is 10 days; that means every 5 days there is an acquisition of one of 
the two satellites, thus providing a high revisit frequency. 

Each Sentinel-2 satellite carries a multispectral instrument with 13 spectral 
channels (in the visible/near-infrared and shortwave infrared spectral range) and 
with 290 km swath width. The Sentinel-2 characteristics are presented in detail in 
[23]. This also applies to level-1 data; level-1C of these products are radiometrically 
and geometrically corrected images with orthorectification and spatial registration 
on a global reference system with sub-pixel accuracy. Since the product size is very 
large, each image is divided into several quadrants in UTM WGS84 projection. 

The average size of a quadrant is 10,980 x 10,980 pixels (rows x columns). For 
visualization, the RGB bands (B04, B03, and B02) were used to generate a quick- 
look quadrant image. For feature extraction, the user can choose different band 
combinations. 


3.3 Third-party mission data 


From the available third-party mission data sets, we selected for demonstration 
four pairs of multi-sensor images of TerraSAR-X and WorldView-2 [27]. 

TerraSAR-X is a German radar satellite launched in June 2007, followed by 
its TanDEM-X twin in 2010. Both operate in X-band and are side-looking SAR 
instruments that offer a wide selection of operating modes and product generation 
options [7]. TerraSAR-X has a revisit cycle of 11 days on the Earth’s equator. We 
selected high-resolution spotlight mode images because they provide the highest- 
resolution data of the target areas. As for the product generation options, we took 
enhanced ellipsoid corrected (EEC) and radiometrically enhanced (RE) data. 
Finally, we took horizontally polarized (HH) or vertically polarized (VV) images, 
as this option is most frequently used. The images have a pixel spacing of 1.25 m and 
a resolution of 2.9 m with WGS-84 map projection. The average size of the images is 
8000 rows x 9600 columns. 

In contrast, World View-2 provides a single panchromatic band and eight 
multispectral bands. It was launched in October 2009 to become a DigitalGlobe 
satellite. The revisit period of the satellite is about 3 days on the Earth’s equator [28]. 
The resolution for the panchromatic band is 0.46 m and for multispectral bands is 
1.87 m. The map projection of WorldView-2 is, again, WGS-84, and the size of these 
images (on average) for panchromatic images is 47,000 x 37,000 pixels (rows x 
columns) and for multispectral images is 11,000 x 9000 pixels (rows x columns). 


4. Typical CANDELA examples 
4.1 Data mining by machine learning 


In EO data mining, a number of researchers have already developed technolo- 
gies for semantic image understanding [29, 30]. The available web engines are 
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focused on the everyday needs of a broad category of users [31]. A very popular 
satellite image data mining system is Tomnod from DigitalGlobe or Google Earth, 
which is targeting general user topics. Especially for EO, there are systems such as 
LandEX [32] which is a land cover management system, while GeoIRIS [33] isa 
system that allows the user to refine a given query by iteratively specifying a set of 
relevant and a set of nonrelevant images. A similar system is IKONA [34] which is 
using relevance feedback in order to analyze the content of very high-resolution EO 
images. Further, the knowledge-driven information mining (KIM) system [41] is 
an example of an active learning system providing semantic interpretation of image 
content. The KIM concept evolved into the TELEIOS prototype [36], complement- 
ing the scope of searching EO images with additional geo-information and in situ 
data. Finally, a cascaded active learning prototype [21] has been integrated into an 
operational EO system [20] to interpret the archives of TerraSAR-X images [37]. 

CANDELA is improving this cascaded active learning system by searching for 
dedicated algorithms for typical Earth observation images. Its implementation, test, 
and validation aim at automated knowledge extraction and image content interpre- 
tation. The targeted performance characteristics are verified for several typical use 
cases and tell us more about the potential of dedicated algorithms with respect to 
general machine learning. 

Figures 4-9 depict typical classification maps for TerraSAR-X and Sentinel-1 
images together with their respective accuracy (e.g., precision/recall) for the cities 
of Venice, Italy, and Munich, Germany. Another example is the Dutch part of the 
Wadden Sea in the Netherlands. The results of the classification map and their accu- 
racy are given in Figures 10 and 11. 


4.2 Data fusion by machine learning 


Currently, what exists in the field of data fusion is a collection of routines/algo- 
rithms that can be linked and embedded for various applications. A very well-known 
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Figure 4. 
TerraSAR-X image of Venice, Italy: (left) a quick-look view of the image and (right) the corresponding 
classification map generated by CANDELA. 
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Figure 5. 

Sentinel-1 image of Venice, Italy (after selecting the area that is covered by TerraSAR-X from the full Sentinel-1 
image): (bottom-left) a quick-look view of the image and (bottom-right) the classification map generated by 
CANDELA. 
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Figure 6. 
Classification accuracy (precision/recall) by comparison between TerraSAR-X (top-left) and Sentinel-1 
(bottom-right) for the Venice image. 
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Figure 7. 
TerraSAR-X image of Munich, Germany: (left) a quick-look view of the image and (right) the classification 
map generated by CANDELA. 
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Figure 8. 
Sentinel-1 image of Munich, Germany (after selecting the area that is also covered by TerraSAR-X): (bottom- 
left) a quick-look view of the image and (bottom-right) the classification map generated by CANDELA. 


open-source toolbox is Orfeo [38] which provides a large number of state-of-the-art 
algorithms to process SAR and multispectral images for different applications. 
Another one is Google Earth [31] that includes a large image database and an expand- 
able number of algorithms that can be used for image processing. 

In our case, we need to recognize different target area details in overlapping SAR 
and multispectral images. For doing this, we selected a number of cities from all over 
the world. The cities are Bucharest in Romania, Munich in Germany, Venice in Italy, 
and Washington in the USA. The selection criteria of these cities were the simultane- 
ous availability of these cities covered by the two satellites and the variety of catego- 
ries that can be found. A difficulty arises when trying to co-align these images, for 
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Figure 9. 
Classification accuracy (precision/recall) by comparison between TerraSAR-X (top-right) and Sentinel-1 
(bottom-left) for the Munich image. 


Agricultural land type 1 


Agricultural land type 2 
Beaches 

Boats 

Deltas / Tidal flats 
Island land 


Lakes 


Medium-density residential areas 


Sea 


Figure 10. 
Sentinel-2 quadrant image of an area of the Dutch Wadden Sea: (left) a quick-look view of the image and 
(right) the classification map generated by CANDELA. 


example, images provided by TerraSAR-X and WorldView-2, because the original data 
have different pixel spacing. To solve this problem, we resampled the panchromatic 


WorldView-2 image in order to co-align it with the TerraSAR-X image [27]. 
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Figure 11. 
Classification accuracy (precision/recall) for the Sentinel-2 quadrant image covering an area of the 
Wadden Sea. 


In the case of Sentinel-1 and Sentinel-2, the images can be rectified and co- 
aligned by publicly available toolbox routines [39]; this allowed us a straightforward 
image comparison. 

While we are accustomed to image fusion as a radiometric combination of 
multispectral images, a comparably mature level of semantic fusion of SAR images 
has not been reached yet. In order to remedy the situation, we propose a semantic 
fusion concept for SAR images, where we combine the semantic image content 
of two data sets with different characteristics. By exploiting the specific imaging 
details and the retrievable semantic categories of the two image types, we obtained 


Figure 12. 
A multi-sensor data set: multispectral image (top-left side), panchromatic image (top-right side), and 
TerraSAR-X image (bottom-center) for the city of Venice, Italy. 
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Classification maps generated using the CANDELA platform for the city of Venice: multispectral image (top- 
left side), panchromatic image (top-right side), TerraSAR-X image (bottom-left side), and fusion of all three 
images (bottom-right side). 


semantically fused image classification maps that allow us to differentiate between 
different categories. 

Figures 12-14 present the classification maps for each sensor and the fused ones 
together with their accuracy (e.g., precision/recall) for the city of Venice, while 
Figures 15-17 apply to the city of Munich. 

For a quantitative assessment, we compared the semantic annotation results 
with the given reference data set and computed precision/recall for each category 
and sensor. Analyzing the figures separately, we observed that the average of 
precision/recall obtained for fused sensor images is higher than the precision/ 
recall of individual sensor images. Unfortunately, there are also cases in which for 
corresponding image patches tiled from different sensor images, the World View-2 
annotations have a different semantic classification when compared to the 
TerraSAR-X results or when a category is missing for one sensor. In our case, in the 
Venice image, the category “buoys” is only detected in the TerraSAR-X image, and 
not in the WorldView-2 image. This has a noticeable impact on the performance of 
the category “boats.” Another example is the category “clouds” that appears in the 
case of the Munich image that is detected in the WorldView-2 image, but not in the 
TerraSAR-X image. 
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Figure 14. 
Classification accuracy (precision/recall) for a selected image taken over the area of Venice using multispectral, 
panchromatic, and SAR images and also the fused image. 


Figure 15. 
A multi-sensor data set: multispectral image (top-left side), panchromatic image (top-right side), and 
TerraSAR-X image (bottom-center) for the city of Munich, Germany. 
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Figure 16. 

Classification maps generated using the CANDELA platform for the city of Munich: multispectral image 
(top-left side), panchromatic image (top-right side), TerraSAR-X image (bottom-left side), and fusion of all 
three images (bottom-right side). 
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Figure 17. 
Classification accuracy (precision/recall) for a selected image over the area of Munich using multispectral, 


panchromatic, and SAR images and also the fused image. 
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5. Data science workflows 


Recently, a new paradigm for Earth observation, namely, Data Knowledge 
Discovery, was introduced [17]. This paradigm defines the entire chain “data- 
information-knowledge-value” and deals with a meaningful EO content extraction, 
i.e., the semantic and knowledge aspects. 

We developed user-invariant and EO domain-specific compensatory methods 
for the individual user- and domain-subjective biases. The derived models generate 
a sharable knowledge body as a means to enable the communication between frag- 
mented knowledge learned from metadata, image data, and other data in synergy 
with the domain expertise of EO users. Today’s EO paradigms and technologies are 
largely domain-oriented and have to support the communication outlined above. 

Artificial intelligence big data in Earth observation [13] forced the development 
of new technologies starting from management platforms [4] and is reaching now 
the information platforms. 

An example for the first category are ESA’s Thematic Exploitation Platforms 
(TEPs) [4] that are designed and focused for coastal applications, forest, geohaz- 
ards, hydrology, polar, urban, and food and security application domains, integrat- 
ing standard processing chains that have low user interaction. The Copernicus 
system (currently still under development) and its data information and access 
services component [18] are a major achievement but still represent a “classic” 
management paradigm. 

Currently, “classic” existing systems/platforms are usually batch-oriented (e.g., 
TEPs, DIAS), but with EOLib [20, 40] and the new CANDELA platform [17], this 
paradigm was “moved” to interactive systems (e.g., supporting active learning). 

There are three perspectives to describe this type of interactive systems: 


e The first one is based on signal-information logic (Figures 18 and 19). 


The objective is the knowledge extraction from the sensor signal of the physically 
meaningful parameters or Earth's surface cover categories. 


The process is divided in two steps: 


o The first step is an automated batch process to manage the satellite image 
product files, i.e., to extract the image data and to select the relevant meta- 
data, to perform a spatial breakdown of the image into patches, to estimate 
for each image patch the particular signatures or primitive descriptors, and 
to further structure the extracted information in a database. 


o In a second step following interactive machine learning paradigms, the 
extracted information is transformed into semantic entities attached to 
each image location. The process is a combination of querying, browsing, 
and active learning. Using positive examples, i.e., training samples for the 
categories of interest and complemented by negative examples to enhance 
the accuracies of each class, a user can define the image semantics adapted to 
a particular application. 


¢ The second perspective is based on the value-adding logic (Figure 20). 
Based on these procedures, value-adding is an iterative process. 


The satellite data are generally multi-mission data, e.g., multispectral and SAR 
data that are restructured in acommon database, which becomes the data source. 
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The signal-information logic scheme: chain > data-information-knowledge. 
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Figure 19. 
The signal-information logic scheme: chain > data-information-knowledge-semantic value. 
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Figure 20. 
The value-adding logic scheme. 


The data preparation component is generating the Analysis-Ready Data (ARD) 
ensuring the least and mandatory processing and organizational steps that enable a 
direct analysis, thus minimizing the user interaction at the data level. 
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Among them are the generation of radiometrically and geometrically calibrated 
data cubes. Browsing the data sets is a first step of visual inspection where the user 
is getting acquainted with the observed structures and their signatures. Further, 
data mining is an automated process to discover the main data particularities and 
categories but also detect artifacts or outliers in the data sets, which are beyond the 
capabilities of human observation, due to the large data volumes and the nonvisual 
nature of the satellite images. The discovered and selected data sets are further ana- 
lyzed in detail by extracting the particular characteristics of the observed scenes 
or objects. The results of the analysis are contributing to update existing models or 
build new models for the observations. Visualization of the model parameters 
or extracted information is a verification step to cope with large complex data 
volumes. Specific evaluation paradigms are needed to build trust in the obtained 
results, to be used to make predictions. The process is iterative, and when new data 
are acquired, they will be analyzed further. 


¢ The third perspective is the implementation architecture logic (Figure 21). 


The implementation of these paradigms requires a concept of integrating 
artificial intelligence with software (SW) system architectures enabling interactive 
multiuser operations in real time relative to the user reaction times. End users will 
be able to work on shared user scenarios, results of their analyses, or information 
extraction procedures. 

The central component is a data index (DI) which is a very specific database 
model for very fast, real-time management, processing, and distribution of large 
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Figure 21. 
The logic implementation architecture scheme. 
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structured and unstructured distributed multi-temporal data sets. The data can be 
efficiently uploaded on demand, coping with large volumes of data from various 
heterogeneous sources. 

The data preparation needs to be able to support various tasks for the ARD 
generation. A workflow orchestration engine will be relaying data and offers 
various processor steps: 


o A deep neural network (DNN module) for physically meaningful feature 
learning 


o Spatiotemporal analysis, e.g., spatiotemporal pattern analysis and extrac- 
tion for understanding the evolution classes, fusing information from 
various sources, not just identifying objects, but in particular spatiotemporal 
patterns and context 


o Data mining to explore heterogeneous multi-temporal data sets. 


The extracted information and data content are again indexed in the DI and pro- 
vided (via web services) to one of the four human-machine interface (HMI) mod- 
ules (i.e., visual browsing, visual analytics, active learning, and event analysis) 
supporting advanced big data visualization and active learning paradigms. Once a 
researcher is satisfied with the results, they can be shared with a restricted group or 
publically via the collaborative layer. These architectures are generically based on 
federated approaches, making it possible to deploy various components where they 
fit best, using cloud technologies and web services for communication. 


6. Conclusions 
The advantages and benefits of the proposed approach are: 


e We do clustering considering the physical parameters behind the sensors 
contrary with the classical classification proposed in AI. 


e With very few examples, we are able to classify the images with high accuracy. 
e Weare able to process multi-sensor data. 


e Weare able to create a semantic scheme adapted to different EO sensors (SAR 
or multispectral), high resolution (e.g., TerraSAR-X or WorldView)/medium 
resolution (e.g., Sentinel-1 or Sentinel-2). 


7. Future work 


During the next years, we expect a wide variety of new satellite image data that 
can be easily downloaded, handled, and analyzed by individual users. We also 
think that a number of new geophysical databases and browse tools will become 
available so that each user has easy access to numerous additional satellite data 
sources together with auxiliary geophysical data from common libraries and data 
management tools supporting in-depth image data analyses and their interpreta- 
tion. Innovative application fields (such as autonomous driving based on machine 
learning and artificial intelligence) will bring us still more data handling tools and 


19 


Advanced Analytics and Artificial Intelligence Applications 


new data archives becoming available via the Internet. In addition, we also suppose 
that these new tools will be supplemented by management and support environ- 
ments, for instance, for system testing and performance monitoring. Within the 
next 5 years, this should result in new established environments for image data 
understanding. 
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